The present invention relates to data processing, and more particularly, to systems and methods of processing data for a data warehouse using in-memory data storage.
Data warehouses are used to store large volumes of data for analysis. Data warehouses receive data from a variety of source systems and make the data available for a wide variety of complex analytic queries. As data is received in a data warehouse, the data may be repeatably stored and transformed as the data is massaged into a form that supports the desired analysis. In some applications, data is received in a very granular level and processed across multiple layers, which may transform, merge, consolidate and aggregate data according to the particular needs of an organization.
According to some architectures, as data is received and processed by a data warehouse, data may be stored in a database management system in one or more data containers. For example, the data warehouse may be a software application (a “data warehouse application”) operating on one or more computers (e.g., servers). At various points in data processing, the data warehouse may interface with a database management system to store data in database tables of a database. Example database tables accessed by a data warehouse application are an active data table and a change log table. An active data table may store values of data that are current at a particular time. A change log table may store a history of how data has changed over time.
One problem with the above architecture for data warehousing is the time delay associated with accessing data stored in tables of a database management system. Each time a new record is received (e.g., from a source system), data is read from the active data table in the database (if the data exists) and loaded into the data warehouse application. Next, incoming data is compared against active data to determine if any changes have occurred. If the data has changed, yet another database management system transaction is performed to write changes to the change log table. These are just examples of what are sometimes referred to as database management system “round trips”, which are time consuming and may delay data processing time in a data warehouse.
The present invention improves data processing in a data warehouse using an in-memory database management system.